Remote trailer backup assist multiple user engagement

ABSTRACT

Systems and methods for operating a vehicle during a remote trailer parking operation include receiving, via a processor, a control instruction from a first mobile device indicative of a first curvature command providing directional control of a vehicle, and a first user engagement indicator. The processor may receive, from a second mobile device, a second control instruction from a second mobile device, the second control instruction can include a second user engagement indicator. The system may determine, based on the first user engagement indicator that first user engagement meets a threshold, and determine, based on the second user engagement indicator that second user engagement meets a threshold. Responsive to determining user engagement with the trailer parking procedure, the processor may cause a vehicle controller to operate the vehicle to park a trailer pivotably disposed with the vehicle based on the first curvature command, the first user engagement indicator, and the second user engagement indicator, to complete a remote trailer parking operation.

BACKGROUND

Operating a vehicle with a trailer in tow may be challenging for somedrivers. This is particularly true for drivers that are not used tobacking up vehicles with attached trailers. Trailer backup assistsystems for vehicles may include an onboard user interface that allowsthe user to steer a trailer towed by the vehicle through an automatedsteering controller that provides the steering motion that moves thetrailer along a user-defined straight path or curvature.

When using a Remote Trailer Maneuver Assist (ReTMA) system, it isdesirable to require a User Engagement signal from the remote device asa confirmation that the user intends vehicle motion. In some cases, thevehicle and/or trailer may be maneuvering in a tight space where thereare tight clearances at multiple locations around the perimeter and/orelevation of the vehicle and/or trailer. In some conventional ReTMAsystems, vehicle sensors establish about a 30 cm virtual bumper tomitigate undesirable contact with an object during parking procedures.However, there currently are no such similar virtual bumpers for thetrailer perimeter or above the vehicle or trailer.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is set forth with reference to the accompanyingdrawings. The use of the same reference numerals may indicate similar oridentical items. Various embodiments may utilize elements and/orcomponents other than those illustrated in the drawings, and someelements and/or components may not be present in various embodiments.Elements and/or components in the figures are not necessarily drawn toscale. Throughout this disclosure, depending on the context, singularand plural terminology may be used interchangeably.

FIG. 1A depicts an example trailer backup assist system according toembodiments of the present disclosure.

FIG. 1B depicts another example trailer backup assist system accordingto embodiments of the present disclosure.

FIG. 1C depicts another example trailer backup assist system accordingto embodiments of the present disclosure.

FIG. 1D depicts another example trailer backup assist system accordingto embodiments of the present disclosure.

FIG. 2 depicts an example computing environment in which techniques andstructures for providing the systems and methods disclosed herein may beimplemented.

FIG. 3 depicts another example remote trailer parking maneuver inaccordance with embodiments of the present disclosure.

FIG. 4 depicts a flow diagram in accordance with the present disclosure.

DETAILED DESCRIPTION Overview

Systems and methods for using multiple devices for ReTMA controlfunctions are described. Disclosed embodiments connect multiple mobiledevices with the ReTMA system for collective monitoring of ReTMA trailermaneuvering operations, and group control of the operation usingwirelessly connected mobile devices.

Three main embodiments are described in accordance with the presentdisclosure. A first embodiment configures mobile devices of one or morespotters operating spotter device(s) disposed at critical locationsaround a remote trailer maneuver operation to monitor and control theoperation in concert with a lead operator device having primary ReTMAcontrol. The spotter device(s) and the lead operator device may sharecontrol of the remote trailer maneuver operation. In one aspect, theowner/administrator of ReTMA may be holding the operator device (alsoreferred to as the administrator device or the lead operator device) andmay configure a ReTMA application operable on the lead operator devicedecide to control vehicle motion and curvature commands. Such controlmay be useful when performing a ReTMA maneuver in the presence ofmultiple obstacles or tight clearances at one or more spots around thevehicle-trailer pair. The lead operator device may administer fullcontrol while receiving images or other control signals from spotterdevices. In other aspects, the lead operator device may assign controlauthority configuring spotter devices to perform aspects of control ofthe trailer maneuver operation such as steering, braking, etc. The leadoperator device may serve as a gatekeeper for ReTMA control signals thatare shared in whole or in part by one or more other connected devices(e.g., the spotter devices).

According to one or more embodiments, any connected smart device (e.g.,a smartphone, fob, or tablet) may be configurable by the system to be aReTMA device enabled to engage in a multiple controller remote controloperation.

In a second embodiment, the cooperative backup assist system mayconfigure multiple remote devices for assistance to teach or train aninexperienced user operating the lead operator device. For example, anowner/administrator of a ReTMA-enabled vehicle and/or trailer may shadowan inexperienced user in training or other environmental situations suchas operations with low-clearance trailer parking situations. In oneaspect, the lead-operator device may transmit a control signal to thespotter device indicative of granted permission for the new user tooperate ReTMA remotely from a 2nd mobile device, such as a smartphone,tablet, fob, wearable smart device, etc., while the lead operator devicemay maintain override capability using the lead operator device. Thesecond embodiment may be analogous to a driver training instructor thatuses a redundant steering wheel, brake or accelerator control inparallel with the driver seat controls being used by a vehicle operatortrainee.

According to a third embodiment, an operator device may increaseobstacle avoidance due to vehicle-identified blind spots and/or tightclearances between the vehicle and/or trailer and obstacles/obstructionsby identifying a blind spot and/or a probability of tight clearancesthat exceeds a threshold, and then generating a message recommendingmore than one spotter. The cooperative backup assist system may alsorecommend a location for the spotter to stand on the ground for anoptimized vantage point (e.g., a particular vehicle or trailer corner,side, front, or vehicle-trailer rear position), suggest that a spotterbe in an elevated position (e.g., on a second floor of a building, acatwalk, etc.), and/or present the location of the spotters on a displayof each spotter device, such as in a map view. In some aspects, thecooperative backup assist system may further suggest an approximatedistance from the vehicle (i.e., to better see overhead obstructions) atwhich the spotter may optimally stand to provide the optimal vantagepoint.

The disclosed methods and system(s) herein may be useful in a trailerreverse system that can assist drivers in operating a vehicle towing atrailer, such as a box trailer or a recreational vehicle (RV) or thelike, in reverse gear without a spotter to help the driver, and withoutadding additional wiring connections or an upgraded onboard controlmodule to the towing or towed vehicle.

These and other advantages of the present disclosure are provided ingreater detail herein. Illustrative Embodiments

The disclosure will be described more fully hereinafter with referenceto the accompanying drawings, in which exemplary embodiments of thedisclosure are shown, and not intended to be limiting.

FIG. 1A depicts an example cooperative backup assist system 107(discussed in greater detail with respect to FIG. 2 ) in an exampleReTMA operating environment 100, according to embodiments of the presentdisclosure. The cooperative trailer backup assist system 107 may beutilized for cooperative control of a towing vehicle 102 to cause atrailer 104 attached to a vehicle 102 to travel along a back-up path 103by allowing a driver of the vehicle 102 (also referred to herein as anoperator or user) to operate an administrator device 108 to specify adesired back-up path 103 shown as a curvature of here (though the pathmay also be a straight line). The cooperative backup assist system 107may provide command control permission to one or more secondary devicesoperated by spotters (second, third, etc. users) that may be positionedat key points to provide additional views of areas that may haveenvironmental features or obstacles to be avoided during the parkingoperation.

The cooperative backup assist system 107 may include a control module110 rigidly disposed onboard the vehicle 102. The control module 110 maybe disposed in communication with one or more sensory devices on thevehicle 102, including for example, one or more side sensory devices113A and 113B, one or more rear vehicle sensors 117, or other sensorydevices not shown. The sensory devices 113A, 113B, and 117 may providesensory input indicative of one or more users, environmental features,and/or obstacles in the ReTMA operating environment 100. The vehiclesensory devices 113A, 113B, and 117 may include LiDAR, RADAR,red-green-blue (RGB) camera systems, infrared camera systems, inertialmeasurement units (IMUs), or other sensory devices. Collectively,vehicle sensory devices 113A, 113B, and 117 are referred to as beingpart of a vehicle sensory system.

In one embodiment, the cooperative backup assist system 107 may includeone or more trailer sensory devices 122, attached to a rear portion ofthe trailer 104, and/or one or more trailer sensory devices 120 disposedon or near one or more trailer side panels or frame members. The vehiclesensory system may be configured and/or programmed to provide sensoryoutput signals indicative of objects, pedestrians, and otherenvironmental features disposed proximate to the trailer 104. Thecontrol module 110 may run an Intelligent Platform Management Bus (IPMB)Communications Protocol. Other protocols are possible, and contemplated.The trailer sensory devices 120 and 122 may include LiDAR, RADAR,red-green-blue (RGB) camera systems, infrared camera systems, inertialmeasurement units (IMUs), or other sensory devices.

The vehicle 102 may be a pickup truck or other towing vehicle equippedwith the trailer backup assist system for controlling the back-up pathof the trailer 104 that is attached to the vehicle 102. Specifically,the vehicle 102 is pivotally attachable to the trailer 104 via a tongue172 longitudinally extending forward from or proximate to the vehiclerear bumper 124. The illustrated trailer 104 is depicted in FIG. 1 asbeing hitched to the vehicle 102 via a trailer hitch 126. It should beappreciated that the vehicle 102 and/or the trailer 104 may take anyshape, form, or configuration.

FIGS. 1B, 1C, and 1D illustrate a trailer backup scenario where thevehicle 102 and trailer 104 are operating in the ReTMA operatingenvironment 100 having one or more obstacle such as, for example, anobstacle 105. The obstacle 105 is illustrated as an exampleenvironmental feature, such as a boulder or structure that may betypical in a trailer parking scenario. As the user/driver 106 operatesthe vehicle 102, they would ideally position themselves at a vantagepoint that provides a view of the obstacle 105, the vehicle 102, thetrailer 104, and a target location or parking destination 119. In FIG.1B, the trailer 104 is positioned at a beginning of the trailer backupmaneuver, and the user/driver 106 has the goal of avoiding the obstacles105, 109 and 111 proximate to the location at which the trailer 104 isto be parked. The example depicted in FIG. 1B further shows a secondReTMA user 112 (also referred to as spotter 112) positioned proximate tothe vehicle 102 and the trailer 104 to perform the trailer parkingoperation that includes navigating the vehicle 102 by parking thetrailer 104 at a parking destination 119 shown between two obstacles 109and 111. During the trailer maneuver operation, the driver 106 may havethe objective to avoid the obstacle 105 as the vehicle 102 maneuvers thetrailer 104 to the parking destination 119 between the two obstacles 109and 111.

A user 106, who is illustrated in FIG. 1 as having a front-right vehicleview of the remote parking operation may have a clear view of one sideof the vehicle 102 and the trailer 104 but may not be positionedsufficiently to see the opposite side of the vehicle 102. Consequently,there may be situations where blind spots during a remote parkingoperation may be disadvantageous to the user 106 of a lead ReTMA mobiledevice 108 as they operate the mobile device 108 to control the vehicle102.

In some aspects, it may be advantageous to have more than one personoperating a connected mobile device act as a spotter, where the second(or more) spotters/spotter devices may assist the driver/user 106 bymonitoring the vehicle 102 and/or trailer 104 during the remote trailerparking operation. For example, as shown in FIG. 1B, the cooperativebackup assist system 107 and/or the user 106 may determine that one ormore blind spots 115 are present in the ReTMA operating environment 100such that a single user from a single viewing position may not have themost advantageous view(s) of all areas of a trailer parking maneuver dueto the presence of one or more obstacle(s) 105, 109, and/or 111. In oneaspect, the cooperative backup assist system 107 may determine that itcan be advantageous for a secondary user (referred to herein as a secondspotter 116), to be positioned such that they have a clear view of onecorner of the vehicle 102 and an obstacle 105 to be avoided during theoperation. The spotter 112 may therefore provide an additional viewingangle that alleviates the blind spot 115. In one aspect, as explained infurther detail hereafter, the cooperative backup assist system 107 maydetermine that there are obstacles, blind spots, or other aspects of theparking maneuver that would decrease a probability of interference withobstacles by positioning the spotter 112 (and a spotter device 114 suchas a connected mobile device) proximate to the blind spot 115. As thevehicle 102 proceeds along the back-up path 103, the driver 106 has aview of the obstacle 105, and the spotter 112 has a view of the obstacle111 and rear-left corner of the trailer 104. The administrator device108 may connect with the spotter device 114 to share command controlresponsibilities such as vehicle starting, stopping, and curvaturecommand that causes the trailer 104 to follow the back-up path 103.

For example, in one aspect, the spotter device 114 and the lead operator(administrator) device 108 may share control of the remote trailermaneuver operation. In one aspect, the owner/administrator of ReTMA (thedriver 106) may control the administrator device 108 and may configure aReTMA application (not shown in FIG. 1B) to be operable on theadministrator device 108 for control of the vehicle motion (start, stop,speed) and curvature (steering that causes the trailer 104 to follow theback-up path 103). Such control may be useful when performing a ReTMAmaneuver in the presence of the multiple obstacles (105, 109, 111, etc.)or tight clearances at one or more spots around the vehicle-trailerpair.

The administrator device 108 may administer full control of the vehicle102 via the control module 110 while receiving images or other controlsignals from the spotter device 114. In other aspects, the administratordevice 108 may assign control authority to the spotter device 118. Thecontrol authority may configure the spotter device 118 to performaspects of control of the trailer maneuver operation such as steering,braking, etc. The administrator device 108 may serve as a gatekeeper forReTMA control signals that are shared in whole or in part by one or moreother connected devices (e.g., the spotter device 114). In some aspects,the driver 106 and the spotter 112 may enable continued vehicleoperation by actuating a user engagement button or feature (not shown inFIG. 1B) that indicates that all connected users 106 and 112 areactively engaged and giving attention to the parking maneuver. If one ormore of the users 106, 108, etc. disengages the user engagement buttonon their respective device, the cooperative backup assist system 107 mayimmediately stop vehicle 102 motion, generate a message indicative ofinstructions for re-engaging the user engagement feature, and continueoperation. This feature also allows for an immediate stop of all vehiclemotion when one or more of the driver 106 and spotter 112 determinesthat a collision may occur, or an adjustment to vehicle speed,curvature, or other control aspects should be altered from a presentstatus.

According to one or more embodiments, any connected smart device (e.g.,a smartphone or tablet) may be configurable by the system to be a ReTMAdevice enabled to engage in a multiple controller remote controloperation.

FIG. 1C depicts the cooperative backup assist system 107 as itconfigures multiple remote devices for assistance to teach or train aninexperienced user operating the lead operator device. For example, theowner/administrator (e.g., the driver 106) of the vehicle 102 and/ortrailer 104 may shadow an inexperienced user (e.g., the spotter 112) intraining or other environmental situations such as operations withlow-clearance trailer parking situations. In one aspect, theadministrator device 108 may transmit a control signal (not shown inFIG. 1C) to the spotter device 114 indicative of granted permission forthe new user (the spotter 112) to operate ReTMA remotely from a spotterdevice 114, while the lead operator device 108 maintains overridecapability using the lead operator device. This feature may be analogousto a driver training instructor that uses a redundant steering wheel,brake or accelerator control in parallel with a vehicle operatortrainee.

For example, the spotter 112 may have limited experience in controllingcurvature command during a trailer backup procedure and over-steer thevehicle 102 to cause the trailer 104 to follow the back-up path 103. Thedriver 106 may observe the over-steering curvature command and overridethe curvature command signal coming from the spotter device 114 using aninterrupt button or other user interface feature (not shown in FIG. 1C)to re-gain control of the curvature command and correct theover-steering error using the administrator device 108. The controlmodule 110 may receive the override signal and immediately implement thenew curvature command that causes the vehicle 102 to accurately followthe back-up path 103 while avoiding the obstacle 105. For example,according to one or more embodiments, the system 107 may provideinstructions that, when executed by one or more processors of arespective device, cause the device to output a human-machine interfacehaving the interrupt button available to the driver 106 such that thedriver may actuate the interrupt button during operation of the vehicle.In other aspects the human-machine interface may provide one or moresettings/controls (not shown in FIG. 1 ) that provide system adjustmentsand settings that, when changed, affect the same change on all or someof the connected devices.

For example, the one or more connected devices may be operated byspotter(s) disposed at one or more locations proximate to the vehicle105. The driver 106 can select a control option using the human-machineinterface (HMI) indicative that location-based enforcement is enabled,where the interface provides one or more user-selectable positions on amap, and accepts indicative user input of desired locations forrespective spotters/spotter devices on the map. Thus, the location ofthe spotter/spotter devices may be viewable by each other via the HMI oneach spotter device. In this example, the spotters can then move withtheir spotter devices to the designated locations. In another example,the driver may be provided a map editing interface using the HMI to drawa perimeter around a particular location, choose a radius magnitudeassociated with a curvature command, etc. This information may then besent to each spotter device for presentation via the HMI, such as in amap view.

According to another aspect, the administrator may set a defaultrequirement for vehicle motion to be associated with, for example, twoor more user engagement signals and/or curvature commands from twoconnected devices but designate one or more particular locations orspotter devices to be exempt from the default requirement.

According to another aspect, the cooperative backup assist system 107may increase obstacle avoidance due to vehicle-identified second blindspot and/or tight clearances between the vehicle 102 and/or trailer 104and additional obstacles/obstructions (e.g., the obstacles 109 and 111)by identifying the blind spot via the vehicle sensory devices 113A,113B, 117, etc., and or the trailer sensory devices 122. The cooperativebackup assist system 107 may evaluate or determine a probability oftight clearances that exceeds a threshold (e.g., less than a thresholddistance of 0.5 meters, 1 meter, etc., between a vehicle or trailersurface and the detected obstacle), and a message recommending more thanone spotter.

The cooperative backup assist system 107 may further recommend alocation for the second (or more) spotters to stand on the ground for anoptimized vantage point (e.g., a particular vehicle or trailer corner,side, front, or vehicle-trailer rear position), or suggest that aspotter be in an elevated position (e.g., on a second floor of abuilding, a catwalk, etc.). As illustrated in FIG. 1C, the cooperativebackup assist system 107 may recommend that a second spotter 116 bepositioned at a rear-right position respective to the trailer 104 toobserve the clearance and curvature of the trailer 104 as it avoids theobstacle 109.

To make this determination, the cooperative backup assist system 107 maydetect the obstacle 105 via the trailer sensory device 122, localize thespotter device(s) 108, 114, and 118 (which may be indicative of aposition of the user operating the spotter devices) determine via thecontrol module 110 that a view to one or more obstacles 105, 111, 109may be obstructed from one or more of the spotter 112, the spotter 116,and/or the driver 106 based on localization made for respective devices108, 114 and/or 118. In some aspects, the control module 110 maydetermine the localized position(s) of the driver 106, the first spotter112, and/or the second spotter 116 via the vehicle sensory system, viaone or more GPS signals received from the devices 108, 114, and/or 118,or via other localization techniques such as Wi-Fi localization,Ultra-Wide Band (UWB), etc., where the sensory systems for the vehicle102, trailer 104, or the mobile device sensory devices (not shown inFIG. 1C) inform the control module 110 of the respective positions ofthe devices 108, 114, 118, respective positions of the users 106, 112,and/or 116, respective positions of the vehicle 102 and/or the trailer104 and respective positions of the obstacles 105, 109, and/or 111.

The control module 110 may include (store on a computer-readable memorynot shown in FIG. 1C) information indicative of physical geometry ofboth the vehicle 102 and the trailer 104, and determine a probability ofclear view of the blind spot based on the vehicle and trailerdimensions, and further based on the respective positions of theconnected devices 108 and 114. An example threshold probability may be20% probability of an unobstructed view, 50% probability of anunobstructed view, etc.

In some aspects, responsive to determining a probability of obstructedview from one or more connected devices such as, for example, theadministrator device 108, the cooperative backup assist system 107 mayfurther suggest an approximate distance from the vehicle 102 or thetrailer 104 at which a spotter (e.g., 112) is ideally positioned toprovide an advantageous view of an obstruction or obstacle. In otheraspects, the cooperative backup assist system 107 may determine thatmore than the current number of connected spotters be used in aparticular parking operation such that the user 106 and spotter team maywork together to control the vehicle 102.

FIG. 1C illustrates a third user (e.g., spotter 116) positioned at anopposite side of the vehicle 102, which may provide a clear view of theobstacle 111. Furthermore, the owner of the vehicle 102 (whom may be,for example, the operator of the administrator device 108) or the system107 may determine that the second blind spot 123 may be alleviated withthe addition of a second spotter 116 to use a spotter device 118 to addcommand control signals to the cooperative parking procedure.

In circumstances similar to the situation depicted in FIGS. 1B and 1C,it may be advantageous for multiple users to provide user engagementand/or curvature commands authorizing vehicle motion with the assistanceof co-operators (e.g., the first spotter 112 operating the spotterdevice 114 and the second spotter 116 operating the spotter device 118)rather than the user 106 stopping or pausing the trailer parking assistoperation and moving around to the location(s) (e.g., the positions ofthe first spotter 112 and/or the second spotter 116), where there aretight clearances present between the vehicle 102 and/or the trailer 104and obstacles 105, 109, and 111. The disclosed system 107 may provide aclear view to blind spots where obstacles are present. It may be anadditional advantage for the cooperative backup assist system 107 todetermine a current position of a user, evaluate the ReTMA operatingenvironment 100 for potential blind spots, and determine that a secondposition offers greater view advantage than a first location of one ormore of the users 106, 112, and/or 116. For example, as shown in FIG.1D, the cooperative backup assist system 107 may determine that it isadvantageous toward the final stage of the remote parking procedure thatthe spotter 116 stand at a second location that provides a view of thetrailer 104 as it becomes proximate to the obstacles 109 and 111. Thecooperative backup assist system 107 may cause the vehicle 102 to stop,generate a message for output on the devices 108, 114, and/or 118indicative of a second (new) position for the spotter device 118 thatprovides the improved vantage point, and indicate an instruction that,when followed, causes the spotter device 118 to observe the final stageof the parking procedure from the new location.

Conventional backup assistance systems may be impractical, as therequired video system for single-user control of the vehicle 102 may ormay not be sufficient to see the opposite vehicle corners or obstaclesthat may be obstructed due to vehicle 102, trailer 104, and/or obstacle105, 109, and/or 111 geometry. To mitigate potential damage to thetrailer 104, it may be advantageous to include two or more connectedwireless mobile devices 108, 114, and 118, etc., which may be configuredand/or programmed to maintain engagement with the parking operation byconnecting with the control module 110 and/or connecting with each otherwirelessly via Bluetooth®, Wi-Fi, or other wireless connection methodswhile providing a clear view of blind spots and obstacles.

As used herein, a spotter may be an individual user providing controlinstructions via a spotting device at strategic vantage points whereobserving (via a camera or other imaging and/or sensory technique, suchas LiDAR, RADAR, etc.) the parking operation can enable the user toidentify one or more potential collisions with objects or features inthe operating environment.

The cooperative backup assist system 107 may provide control capabilityto halt the trailer backup maneuver at any time, via any connectedspotting device (e.g., device(s) 108, 114, and/or 118). The proposedarchitecture of the cooperative backup assist system 107 can improveReTMA control capability by using multiple devices 108, 114, and 118,etc., which may be operated by the plurality of secondary users (e.g.,spotter devices) 114, 118, etc., to control the vehicle 102 and performthe trailer parking operation while reducing the likelihood ofcollisions with the obstacles 105 and 107.

The trailer backup assist system may be configured to steer the vehicle102 automatically or manually to guide the trailer 104 on the desiredcurvature or back-up path as a driver (e.g., the user 106) operates thecooperative backup assist system 107 to control the operating speed andcurvature path for the vehicle 102. The cooperative backup assist system107 may monitor the dynamics of the trailer 104 using sensory outputfrom the vehicle sensory devices 113A, 113B and/or 117, and/or trailersensory output from the trailer sensory devices 120 and/or 122, such asa yaw rate, and communicates with the control module 110 disposedonboard the vehicle 102. The trailer backup assist system 107, accordingto such an embodiment, may also include a vehicle sensor system (e.g.,similar or identical to the vehicle sensory system 294 as shown withrespect to FIG. 2 ) that can generate information used for navigationsuch as, for example, a vehicle yaw rate and a vehicle speed.

Prior to discussing further embodiments of the present disclosure ingreater detail, a brief discussion of system components andfunctionality shall be discussed with respect to FIG. 2 .

FIG. 2 depicts an example computing environment 200 that can include thevehicle 205 operating as part of a cooperative backup assist system 207.The cooperative backup assist system 207 may be substantially similar oridentical to the cooperative backup assist system 107 described withrespect to FIGS. 1A-1D. The vehicle 205 may include an automotivecomputer 245, a vehicle sensory system 294 that can include RADAR,LiDAR, camera, (IMUs), and other sensory devices, and a Vehicle ControlUnit (VCU) 265 that typically includes a plurality of electronic controlunits (ECUs) 217 disposed in communication with the automotive computer245 and backup assist system 207 as shown in FIGS. 1C and 1D.

A mobile device 220, which may be associated with a user 106 and thevehicle 102, may connect with the automotive computer 245 using wiredand/or wireless communication protocols and transceivers (not shown inFIG. 2 ). The mobile device 220 may be substantially similar oridentical to the devices 108, 114, and 118 as described with respect toFIGS. 1A-1D, and/or devices 308, 314, and 318 as described with respectto FIG. 3 . The mobile device 220 may be communicatively coupled withthe vehicle 205 via one or more network(s) 225, which may communicatevia one or more wireless channel(s) 230, and/or may connect with thevehicle 102 and/or other connected mobile devices (such as the devices114 and/or 118 as shown in FIG. 1 ) using near field communication (NFC)protocols, Bluetooth® protocols, Wi-Fi, UWB, and other possiblelocalization, device connection and communication, and data sharingtechniques. The vehicle 102 may also receive and/or be in communicationwith a Global Positioning System (GPS) 275.

The automotive computer 245 may be or include an electronic vehiclecontroller having one or more processor(s) 250 and memory 255. Theautomotive computer 245 may, in some example embodiments, be disposed incommunication with the mobile devices 108, 114 and/or 118, and one ormore server(s) 270. The server(s) 270 may be part of a cloud-basedcomputing infrastructure, and may be associated with and/or include aTelematics Service Delivery Network (SDN) that provides digital dataservices to the vehicle 102.

Although illustrated as a pickup truck, the vehicle 205 may take theform of another passenger or commercial automobile such as, for example,a car, a crossover vehicle, a sport utility, a van, a minivan, a taxi, abus, etc., and may be configured to include various types of automotivedrive systems. Exemplary drive systems can include various types ofinternal combustion engine (ICE) powertrains having a gasoline, diesel,or natural gas-powered combustion engine with conventional drivecomponents such as, a transmission, a drive shaft, a differential, etc.In another configuration, the vehicle 205 may be configured as anelectric vehicle (EV). More particularly, the vehicle 102 may include abattery EV (BEV) drive system, or be configured as a hybrid EV (HEV)having an independent onboard power plant, a plug-in HEV (PHEV) thatincludes a HEV powertrain connectable to an external power source,and/or includes a parallel or series hybrid powertrain having acombustion engine power plant and one or more EV drive systems. HEVs mayfurther include battery and/or super capacitor banks for power storage,flywheel power storage systems, or other power generation and storageinfrastructure. The vehicle 205 may be further configured as a fuel cellvehicle (FCV) that converts liquid or solid fuel to usable power using afuel cell, (e.g., a hydrogen fuel cell vehicle (HFCV) powertrain, etc.)and/or any combination of these drive systems and components.

Further, the vehicle 205 may be a manually driven vehicle, and/or beconfigured to operate in a fully autonomous (e.g., driverless) mode(e.g., level-5 autonomy) or in one or more partial autonomy modes.Examples of partial autonomy modes are widely understood in the art asautonomy Levels 1 through 5. An autonomous vehicle (AV) having Level-1autonomy may generally include a single automated driver assistancefeature, such as steering or acceleration assistance. Parking assistsystems may be included as one such Level-1 autonomous system. Adaptivecruise control is another example of a Level-1 autonomous system thatcan include aspects of both acceleration and steering. Level-2 autonomyin vehicles may provide partial automation of steering and accelerationfunctionality, where the automated system(s) are supervised by a humandriver that performs non-automated operations such as braking and othercontrols. Level-3 autonomy in a vehicle can generally provideconditional automation and control of driving features. For example,Level-3 vehicle autonomy typically includes “environmental detection”capabilities, where the vehicle can make informed decisionsindependently from a present driver, such as accelerating past aslow-moving vehicle, while the present driver remains ready to retakecontrol of the vehicle if the system is unable to execute the task. Thecooperative backup assist system 207 may further include Level-3autonomy features. Level-4 autonomy includes vehicles having high levelsof autonomy that can operate independently from a human driver, butstill include human controls for override operation. Level-4 automationmay also enable a self-driving mode to intervene responsive to apredefined conditional trigger, such as a road hazard or a system event.Level-5 autonomy is associated with autonomous vehicle systems thatrequire no human input for operation, and generally do not include humanoperational driving controls.

According to embodiments of the present disclosure, the cooperativebackup assist system 207 may be configured to operate with a vehiclehaving a Level-1 through Level-5 autonomous vehicle controller (notshown in FIG. 2 ).

The mobile device 220 generally includes a memory 223 for storingprogram instructions associated with an application 235 that, whenexecuted by a mobile device processor 221, performs aspects of thedisclosed embodiments. The application (or “app”) 235 may be part of thecooperative backup assist system 207, or may provide information toand/or receive information from the cooperative backup assist system207.

In some aspects, the mobile device 220 may communicate with the vehicle102 through the one or more wireless channel(s) 230, which may beencrypted and established between the mobile device 220 and a TelematicsControl Unit (TCU) 260. The mobile device 220 may communicate with theTCU 260 using a wireless transmitter (not shown in FIG. 2 ) associatedwith the TCU 260 on the vehicle 102. The transmitter may communicatewith the mobile device 220 using a wireless communication network suchas, for example, the one or more network(s) 225. The wireless channel(s)230 are depicted in FIG. 2 as communicating via the one or morenetwork(s) 225, and via one or more direct channel(s) 233. The directwireless channel(s) 233 may include various low-energy protocolsincluding, for example, Bluetooth®, BLE, or other Near FieldCommunication (NFC) protocols. For example, the system 307 may sendand/or receive command control signals between connected devices 108,114, and/or 118, and from and to the control module 110.

The network(s) 225 illustrate an example of a communicationinfrastructure in which the connected devices discussed in variousembodiments of this disclosure may communicate. The network(s) 225 maybe and/or include the Internet, a private network, public network orother configuration that operates using any one or more knowncommunication protocols such as, for example, transmission controlprotocol/Internet protocol (TCP/IP), Bluetooth®, Wi-Fi based on theInstitute of Electrical and Electronics Engineers (IEEE) standard802.11, Ultra-Wide Band (UWB), and cellular technologies such as TimeDivision Multiple Access (TDMA), Code Division Multiple Access (CDMA),High Speed Packet Access (HSPDA), Long-Term Evolution (LTE), GlobalSystem for Mobile Communications (GSM), and Fifth Generation (5G), toname a few examples.

The automotive computer 245 may be installed in an engine compartment ofthe vehicle 102 (or elsewhere in the vehicle 102) and operate as afunctional part of the cooperative backup assist system 207, inaccordance with the disclosure. The automotive computer 245 may includeone or more processor(s) 250 and a computer-readable memory 255.

The one or more processor(s) 250 may be disposed in communication withone or more memory devices disposed in communication with the respectivecomputing systems (e.g., the memory 255 and/or one or more externaldatabases not shown in FIG. 2 ). The processor(s) 250 may utilize thememory 255 to store programs in code and/or to store data for performingaspects in accordance with the disclosure. The memory 255 may be anon-transitory computer-readable memory. The memory 255 can include anyone or a combination of volatile memory elements (e.g., dynamic randomaccess memory (DRAM), synchronous dynamic random access memory (SDRAM),etc.) and can include any one or more nonvolatile memory elements (e.g.,erasable programmable read-only memory (EPROM), flash memory,electronically erasable programmable read-only memory (EEPROM),programmable read-only memory (PROM), etc.)

The VCU 265 may share a power bus 278, and may be configured tocoordinate the data between vehicle 102 systems, connected servers(e.g., the server(s) 270), and other vehicles (not shown in FIG. 2 )operating as part of a vehicle fleet. The VCU 265 can include orcommunicate with any combination of the ECUs 217, such as, for example,the backup control module 110 as shown in FIG. 1 , a Body Control Module(BCM) 293, a Transmission Control Module (TCM) 290, the TCU 260, aRestraint Control Module (RCM) 287, etc. In some aspects, the VCU 265may control aspects of the vehicle 102, and implement one or moreinstruction sets received from the application 235 operating on themobile device 220, from one or more instruction sets received from thecooperative backup assist system 207, and/or from instructions receivedfrom an AV controller (not shown in FIG. 2 ).

The TCU 260 can be configured to provide vehicle connectivity towireless computing systems onboard and offboard the vehicle 102 such asthose disposed in the mobile device 220, and may include a Navigation(NAV) receiver 288 for receiving and processing a GPS signal from theGPS 275, a Bluetooth® Low-Energy (BLE) Module (BLEM) 295, a Wi-Fitransceiver, an Ultra-Wide Band (UWB) transceiver, and/or other wirelesstransceivers (not shown in FIG. 2 ) that may be configurable forwireless communication between the vehicle 102 and other systems,computers, and modules, such as spotting devices 108, 114 and 118. TheTCU 260 may be disposed in communication with the ECUs 217 by way of abus 180. In some aspects, the TCU 260 may retrieve data and send data asa node in a CAN bus.

The BLEM 295 may establish wireless communication using Bluetooth® andBluetooth® Low-Energy® communication protocols by broadcasting and/orlistening for broadcasts of small advertising packets, and establishingconnections with responsive devices that are configured according toembodiments described herein. For example, the BLEM 295 may includeGeneric Attribute Profile (GATT) device connectivity for client devicesthat respond to or initiate GATT commands and requests, and connectdirectly with the mobile device 220.

The bus 180 may be configured as a Controller Area Network (CAN) busorganized with a multi-master serial bus standard for connecting two ormore of the ECUs 217 as nodes using a message-based protocol that can beconfigured and/or programmed to allow the ECUs 217 to communicate witheach other. The bus 180 may be or include high-speed CAN (which may havebit speeds up to 1 Mb/s on CAN, 5 Mb/s on CAN Flexible Data Rate (CANFD)), and can include a low-speed or fault tolerant CAN (up to 125Kbps), which may, in some configurations, use a linear busconfiguration. In some aspects, the ECUs 217 may communicate with a hostcomputer (e.g., the automotive computer 245, the cooperative backupassist system 207, and/or the server(s) 270, etc.), and may alsocommunicate with one another without the necessity of a host computer.The bus 180 may connect the ECUs 217 with the automotive computer 245such that the automotive computer 245 may retrieve information from,send information to, and otherwise interact with the ECUs 217 to performsteps described according to embodiments of the present disclosure. Thebus 180 may connect CAN bus nodes (e.g., the ECUs 217) to each otherthrough a two-wire bus, which may be a twisted pair having a nominalcharacteristic impedance. The bus 180 may also be accomplished usingother communication protocol solutions, such as Media Oriented SystemsTransport (MOST) or Ethernet. In other aspects, the bus 180 may be awireless intra-vehicle bus.

The VCU 265 may control various loads directly via the bus 180communication or implement such control in conjunction with the BCM 293.The ECUs 217 described with respect to the VCU 265 are provided forexemplary purposes only, and are not intended to be limiting orexclusive. Control and/or communication with other control modules notshown in FIG. 2 is possible, and such control is contemplated.

In an example embodiment, the ECUs 217 may control aspects of vehicleoperation and communication using inputs from human drivers, inputs froman autonomous vehicle controller, the cooperative backup assist system207, and/or via wireless signal inputs received via the wirelesschannel(s) 233 from other connected devices such as the mobile device220, among others. The ECUs 217, when configured as nodes in the bus180, may each include a central processing unit (CPU), a CAN controller,and/or a transceiver (not shown in FIG. 2 ). For example, although themobile device 220 is depicted in FIG. 2 as connecting to the vehicle 102via the BLEM 295, it is possible and contemplated that the directwireless channel(s) 233 may also or alternatively be established betweenthe mobile device 220 and one or more of the ECUs 217 via the respectivetransceiver(s) associated with the module(s).

The BCM 293 generally includes integration of sensors, vehicleperformance indicators, and variable reactors associated with vehiclesystems, and may include processor-based power distribution circuitrythat can control functions associated with the vehicle body such aslights, windows, security, door locks and access control, and variouscomfort controls. The BCM 293 may also operate as a gateway for bus andnetwork interfaces to interact with remote ECUs (not shown in FIG. 2 ).

The BCM 293 may coordinate any one or more functions from a wide rangeof vehicle functionality, including energy management systems, alarms,vehicle immobilizers, driver and rider access authorization systems,Phone-as-a-Key (PaaK) systems, driver assistance systems, AV controlsystems, power windows, doors, actuators, and other functionality, etc.The BCM 293 may be configured for vehicle energy management, exteriorlighting control, wiper functionality, power window and doorfunctionality, heating ventilation and air conditioning systems, anddriver integration systems. In other aspects, the BCM 293 may controlauxiliary equipment functionality, and/or be responsible for integrationof such functionality.

The computing system architecture of the automotive computer 245, VCU265, and/or the cooperative backup assist system 207 may omit certaincomputing modules. It should be readily understood that the computingenvironment depicted in FIG. 2 is one example of a possibleimplementation according to the present disclosure, and thus, it shouldnot be considered limiting or exclusive.

FIG. 3 depicts a top view of a remote trailer parking maneuver, inaccordance with embodiments of the present disclosure. FIG. 3illustrates another embodiment using a connected vehicle 302 towing atrailer 304, where the vehicle 302 and trailer 304 are equipped with acooperative backup assist system 307. The vehicle 302 may besubstantially similar or identical to the vehicles 102 and 205, thetrailer 304 may be substantially similar or identical to the trailers104 and 204, and the cooperative backup assist system 307 may besubstantially similar or identical to the systems 107 and 207 describedwith respect to FIGS. 1A-1D and FIG. 2 . Moreover, the users 306, 312,and 316 may be similar to the users described in prior figures, as wellas the respective devices 308, 314, and 318. The cooperative backupassist system 307 may aggregate curvature command inputs from themultiple connected devices.

The cooperative backup assist system 307 may use one or more of thefollowing methods, which may be user-selectable using the administratordevice 308. According to a first method, the cooperative backup assistsystem 307 may control the vehicle 302 using a device 308 for curvatureinput only, where the cooperative backup assist system 307 configuresone or more of the rest of the connected device(s), for example, devices314 and 318, for individual engagement input only.

In another example, the cooperative backup assist system 307 may use asecond method where one device (e.g., the administrator device 308) isused for curvature input plus individual engagement inputs thatdemonstrate the user’s 306 attention on the remote parking procedure,where the cooperative backup assist system 307 configures the otherdevice(s) 314, 318 for individual engagement input only. For example,the system 307 may determine that one or more devices 308, 314, or 318have not sent a command control signal indicative that a respective useris actively actuating the user engagement feature (button, gesture,etc.) as described in prior embodiments. Responsive to determining thatone of the connective users may not be engaged in the trailer parkingprocedure, the system 307 may cause the vehicle to stop motion until allconnected users are actively engaged.

According to a third embodiment, a consensus-based engagement iscontemplated where two or more devices agree of the three connecteddevices 308, 314, and/or 318. The cooperative backup assist system 307may determine, within a threshold of difference between control inputsreceived from the devices 308, 314, and/or 318, to engage a particularvehicle control action. In some aspects, the group engagement criterionmay be set to have the respective curvature inputs agree within achangeable tolerance of difference with each other. By including arequirement for agreement on a vehicle control action, the cooperativebackup assist system 307 may determine a higher likelihood orprobability that the control instruction will result in the vehicleand/or trailer operating on a clear path.

According to an embodiment, the cooperative backup assist system 307 mayreconcile slight curvature input differences by averaging the curvatureinputs, or by weighting a particular curvature input according toadministrative authority given to the lead operator device. Responsiveto determining that any user device has generated a vehicle controlinstruction that adjusts a vehicle or trailer path, the cooperativebackup assist system 307 may cause a human-machine interface (HMI)operating on one or more of the connected devices to share an updatethat a vehicle path adjustment was made by displaying an output colorchange and/or by generating an audible or haptic feedback output via thedevice(s) 308, 314, and/or 318. An illustrative HMI 320 associated withspotter device 314 is provided, showing in a map view the position ofthe vehicle, trailer and spotter devices, including an indicationhighlighting of the position of the spotting device on which theinterface is displayed.

In some aspects, the cooperative backup assist system 307 may stopvehicle 302 motion when either of the following occur: (1) one or moreof the connected devices 308, 314, and/or 318 determines that arespective user 306, 312, and/or 316 has lifted their finger off thecurvature input area (not shown in FIG. 3 ) of a mobile device HMI; (2)one or more of the connected devices 308, 314, and/or 318 determinesthat the respective user has input a curvature command from a positionthat exceeds a predetermined threshold of distance from one or moreother connected mobile devices; and (3) one or more of a set ofestablished ReTMA stopping criteria are met (e.g., the system determinesthat an object encroachment within 30 cm of towing vehicle is imminent,the system determines an unlock trigger is engaged, the systemdetermines that a vertical angle of the towing vehicle or the trailerhas exceeded a threshold for vertical angle, etc.).

According to one or more embodiments, the cooperative backup assistsystem 307 may configure the connected device(s) 308, 314, and/or 318via the application 235 (shown in FIG. 2 ) for interfaces withinertial-sensor-based curvature requests. For example, the HMI operativeon a mobile device may include a curvature and an engagement interfaceelement.

The group engagement criterion may require that curvature inputsrespectively made to a connected mobile device (e.g., 314) be within athreshold tolerance of difference from any other mobile device curvatureinput. The group engagement criterion may indicate that there exists aconsensus of users 108, 112, and 114 on a clear path (e.g., the path 103as shown in prior figures) for the vehicle 302 and trailer 304, or thatthere is a lack of consensus of connected mobile device users for thecontrol input that provides the clear path for the vehicle and thetrailer. In one aspect, the cooperative backup assist system 307 mayreconcile slight curvature input differences within the tolerance byaveraging them, or by giving priority to the administrator device.

For example, with reference again to FIG. 2 , the processor(s) 250 mayreceive localization data from the vehicle sensory system 294 indicativeof one or more localized positions for obstacles and users proximate tothe vehicle 205. The system 207 may determine that the mobile device 220is the administrator device (by evaluating a user-selectableidentification (not shown in FIG. 2 ) indicating the administratordevice among two or more connected devices), and compare a curvaturecommand from the administrator device 220 to a curvature commandreceived from one or more secondary devices (e.g., the spotter devices114 and/or 118 as shown in FIGS. 1A-1D).

The system 207 may average the curvature commands received from allconnected devices, average the difference between the commands (e.g., bymeasuring an angle, distance, deviation from a planned path, or anothermetric), and compare the averaged difference to a predeterminedthreshold of curvature command difference. An example may be 2 linearfeet of distance if the metric being evaluated is distance. Anotherexample may be 10 degrees distance if the metric being measured iscurvature angle. Other metrics are possible and contemplated in thisdisclosure.

Returning again to FIG. 3 , the system 307 may determine that thedifference between one or more curvature commands exceeds the average ofthe curvature commands, and cause the vehicle 302 to change a vehicleoperation setting responsive to determining that the difference exceedsthe threshold. In some aspects, the cooperative backup assist system 207may stop the vehicle 302 motion responsive to determining that one ormore triggers have occurred. For example, a first trigger may includeone or more of the connected devices 308, 314, and/or 318, determiningthat a respective user (one of the users 306, 312, and/or 316) haslifted their finger off the curvature input area of the mobile deviceHMI, which may indicate a lack of user engagement to the trailermaneuver operation.

In another aspect, a second trigger may include one or more of theconnected devices 308, 314, and/or 318, determining that a user hasexecuted a curvature command that exceeds the threshold of differencefrom one or more curvature commands executed by the other mobiledevices. In yet another aspect, a third trigger may include determiningthat normal ReTMA stopping criteria are met (e.g.., the system detectsan object encroachment within 30 cm of towing vehicle, the systemdetects an unlock trigger, or the system detects that the vehicle or thetrailer has exceeded a vertical angle threshold, etc.).

In another aspect, the user interface may allow the administrator deviceto require that the other connected mobile devices provide the requiredinput for vehicle motion and curvature. The HMI may present thisselectable option from a menu selection element on a screen, by a hardbutton in the vehicle, by receiving a user voice command, or anothertype of user input or selection.

In one or more embodiments, the cooperative backup assist system 307 mayidentify devices capable of providing an independent user engagementsignal to the administrator device 108, and provide a user selectableoutput that, when engaged, selects one or more of the connected mobiledevices needed for vehicle motion. A user engagement signal may indicatethat a user operating the sending mobile device (e.g., device 314) isactively actuating a user engagement button (not shown in FIG. 3 ),performing a complex gesture indicative that the user is engaged withthe trailer maneuver operation, or providing another user engagementindication that may be characterized or measured using a thresholdvalue. For example, a spotter device 314 may send, to the administratordevice 308, a first user engagement signal that meets a predetermineduser engagement threshold having a value of 1 (engaged or actuated).

An example of a first user engagement signal that does not meet apredetermined user engagement threshold is a user engagement value of 0(not engaged or actuated). A non-engaged or actuated signal may be asignal responsive to the user of device 314 taking his/her finger off ofthe user engagement actuation button. The first user engagementthreshold may be actuation of a user engagement button by the user onthe spotter device 314. The cooperative backup assist system 207 mayfurther allow the lead mobile device to add new spotter mobile devicesdeices.

In some aspects, the cooperative backup assist system 307 may provide aninterface for the administrator to select how to enforce multipledevices. For example, multiple spotter devices 316 and/or 314 may bedirected at certain locations, such as the blind vehicle corner asoccupied by the second user 312. In this case, the administrator device308 can receive a user selection or a location-based enforcement and canselect the respective spotter location(s) on a map.

In another aspect, the administrator device 308 may include an interfaceusable for receiving user input indicative of a geofence or perimeterthat localizes a position for the spotter device, chooses a radiusmagnitude, and/or other selectable/definable options.

The cooperative backup assist system 207 may provide one or moreenforcement rules that can be based on the remote device. For example,some devices may be configured and/or programmed to command vehiclemotion when another device 314 and/or 318 also provides a userengagement signal and/or curvature command. This may be desirable ifcertain people are less experienced with ReTMA or trailering. In turn,the lead device may designate that the rule be relaxed once a secondmobile device has been used for a minimum amount of time.

According to another aspect, the cooperative backup assist system 307may configure one or more devices 308, 314, and/or 318 with an interface(interface not shown in FIG. 3 ) allowing the administrator device 308to define a minimum number of connected devices. For example, theadministrator device 308 may be further programmed and/or configured toreceive one or more inputs indicative of rule exceptions. For example,the administrator may set the default requirement for vehicle motion tobe user engagement signals and/or curvature commands from two devices314, 318, but designate certain locations or devices to be exempt. Adevice being exempt may be based on the device meeting or satisfying oneor more exemption criteria or parameters, such as include a userselectable option that indicates that another device is permitted toexceed predetermined thresholds or provide curvature commands thatoverride the administrator device. In another example of an exemptioncriteria or parameter, a geographically bound area may be indicated tothe system that is exempt from being considered as a potential blindspot, or exempted from being considered as relevant to the parkingprocedure.

Furthermore, the administrator device 108 may be configured and/orprogrammed to designate a hierarchy of connected mobile devices. Forexample, the cooperative backup assist system 307 may define one or moreareas of operation indicating that vehicle and/or trailermaneuverability may be so difficult to maneuver that they always requiremultiple devices 314, 318, even if some of the devices would normally beallowed to operate independently. Similarly, the cooperative backupassist system 307 may direct that the mobile device have another remotedevice that also provides a user engagement signal and curvature commandto enable vehicle motion.

In another embodiment of the present disclosure, the cooperative backupassist system 307 may provide a user engagement signal that isdesignated as from an administrator device. For example, the cooperativebackup assist system 307 may designate a minimum number of devices byevaluating, via the vehicle sensory system 294 a relative location of anobstacle or user, determine one or more blind spots based on vehicle andtrailer geometry, user device positions, and localized obstructions, anddetermine one mobile device of the multiple connected mobile devices tobe the administrator device based on having an optimal vantage point.

In another aspect, the system 307 may define particular devices toprovide a user engagement signal and/or curvature command to allowvehicle motion. For example, using the exemption procedures above, thesystem 307 may designate devices 308 and 318 to provide curvaturecommand, while exempting the device 314 from providing curvaturecommand. Furthermore, the vehicle 102 may receive sensory data from thevehicle sensory system 294 indicating relative locations of the devices108, 114, 118, or may receive localization signals from those devicesthat indicate their position, and require the respective devices 108,114, and/or 118 to be spaced a minimal distance apart such that there isoptimal coverage from the recommended vantage points. In another aspect,the system 307 may recommend that the devices (one or more of 108, 112,116) be located in zones to achieve a minimal visual coverage,respectively, by the users (the driver 106, spotter 112, and spotter116) as described in prior embodiments

When one device (e.g., the administrator device 308) stops providing auser engagement signal or the curvature commands deviate beyond acurvature command threshold, the other connected devices 314 and/or 318may alert the users 306, 312, and 316 that the vehicle motion has beenstopped, and advise which device(s) needs to provide a user engagementsignal to enable vehicle motion and if the curvature commands deviatedtoo much. Using the system of FIG. 2 for explanation, the system 207 maydetermine, via the processor(s) 250, that the user engagement signalindicates a lack of actuation, and cause the mobile device 220 (and moreparticularly, the application 235) to output the advice via the mobiledevice HMI. Since the administrator is not providing a user engagementsignal, the administrator will be able to view information of interestusing his/her remote device. For example, the administrator may view thelocation and location history of the vehicle, trailer, and devicesproviding the user engagement signal.

Referring again to FIG. 3 , the cooperative backup assist system 307 mayfurther configure the administrator device 308 also view data from thevehicle 102 such as occupant locations and/or perception sensor videowith object distance and classification information. By contrast,responsive to determining that one of the connected devices providing auser engagement signal is an administrator device, and/or determiningthat either of the connected devices stops providing a user engagementsignal, the cooperative backup assist system 307 may generate an outputmessage requesting that the respective user agree to act as theadministrator device. The administrator may then choose which device(s)may command vehicle motion independently. Furthermore, the administratordevice 308 may be programmed and/or configured to override minimaldistance, specific zone requirements, or curvature command deviationsfor the administrator device 308.

In another embodiment, the vehicle 302 may identify that an area (e.g.,the space between the obstacles 309 and 311) has tight clearances basedon the known dimensions of the vehicle and trailer as compared to theenvironment. The environment may be sensed directly using the vehicle302 and/or trailer 304 perception sensors (not shown in FIG. 3 ) or anobstacle map may be retrieved from a geographic database stored on thevehicle or in the cloud. For example, with reference to FIG. 2 , theserver(s) 270 may store an obstacle map (not shown in FIG. 2 )indicative of a perceived field of obstacles, obstructions, andenvironmental features associated with a known geographic area in theoperating environment. In one example, the system 207 may create such amap after a connected vehicle 205 operates in the environment, transmitthe obstacle map to the server(s) 270, and the server(s) 270 may storethe obstacle map for future use by the vehicle 205 or another connectedvehicle operating in the same environment at a later time. The vehiclemay also identify areas around the vehicle and trailer where there isnot direct sensor coverage.

Referring again to FIG. 3 , the vehicle 302 may suggest the number ofdevices and location zones around the vehicle 102 and trailer 104, tooperate ReTMA with optimal user visual coverage around the vehicle andtrailer. For example, the cooperative backup assist system 307 maydetermine location of objects in the operating environment using areal-time or historical map, image, sensory device information, or otherstored vehicle perception input, and determine the number of devicescurrently configured for use as spotter devices disposed proximate tothe vehicle 302 and/or the trailer 304. The cooperative backup assistsystem 307 may do this by localizing the spotter devices 316 and/or 318using UWB, Bluetooth®, GPS, or other localization techniques.

The cooperative backup assist system 307 may further localizeenvironmental features or obstacles using the sensory data, historicalor real-time images, etc., using one or more known techniques. Forexample, the cooperative backup assist system 207 may receive, from oneor more vehicle sensory devices or trailer 304 sensory devices, such asa LiDAR device, a RADAR device, vehicle camera sensor, or other type ofsensory device that indicates presence and relative location of anenvironmental feature or obstacle to be avoided during the trailerbackup maneuver.

The cooperative backup assist system 307 may determine potentialprobabilities for collisions based on known dimensions of the trailer304, the vehicle 302, a curvature (or approach) angle, etc. For example,the cooperative backup assist system 307 may determine, based on thelocalized environmental features, obstacles (e.g., 309 and 311), andposition(s) of the spotter devices one or more “tight spots” or higherprobability areas for vehicle 302 or trailer 304 collisions where thepotential collision may be mitigated by positioning or repositioning theconfigured spotter devices or by adding additional spotter devices.

To make this determination, the cooperative backup assist system 307 maydetermine one or more positions for devices that may improve systemand/or user perception of how the vehicle 302 and trailer 304 navigatesaround the localized obstacle or feature.

According to the use case, where multiple devices 308, 314, and/or 318can be used as part of a training exercise, the cooperative backupassist system 307 may allow each connected mobile device (e.g., 308,314, 318) to maintain the lead mobile device control privileges andbehavior as noted above. Such a scenario was explained in detail withrespect to FIG. 1C. The cooperative backup assist system 107 mayconfigure the device(s) 308, 314, 318 such that any respective mobiledevice may only: (1) allow all users to have control privileges but theAdmin phone can override/trump any one of the other phones, (2) allowthe admin to actually control the vehicle while providing secondaryusers with a mirror image of what the admin is doing, or (3) allow a mixof the above where, for example, only one of two additional spotters isassigned trainer mode but the other spotter is assigned full control.

For yet another use case, the cooperative backup assist system 307 mayalso recommend the best location for the spotter to stand on the groundfor the best vantage point (e.g., which corner, side, front, or rear ofthe vehicle-trailer unit), or it may suggest that for optimum viewingone spotter should be in an elevated position (e.g., 2nd floor of abuilding, catwalk, etc.), or the cooperative backup assist system 107may suggest an approximate distance from the vehicle that providesoptimum viewing of potential collision points or distance constrictions(e.g., to better see overhead obstructions). The cooperative backupassist system 107 may suggest use of an unmanned aerial vehicle (e.g., adrone) to achieve an optimum vantage point. For example, if thecooperative backup assist system 107 determines that an obstacle at arear right side of a trailer is particularly “tight” or in closeproximity to a projected curvature angle needed to complete the trailerparking maneuver, the cooperative backup assist system 307 may recommendthat a spotter positioned in another location or move to the rear rightside of the trailer to provide a clear view of the obstacle to avoid.

In another aspect, when all spotters (and spotter devices) currentlyconfigured to participate with the administrator device are in optimalpositions such that their repositioning would leave an advantageousviewing angle unviewed by a spotter device, the cooperative backupassist system 307 may recommend adding one or more additional spotterdevices, and may generate a graphic or verbal indication of therecommended position of the newly added spotter device. For example, thecooperative backup assist system 307 may generate an output indicativeof the phrase “Consider Adding A Spotter Device At the Rear Right SideOf Trailer” or the like.

Referring to FIG. 4 at step 405, the method 400 may commence withreceiving, via a processor, a control instruction from a first mobiledevice indicative of a first curvature command providing directionalcontrol of a vehicle, and a first user engagement indicator. The firstuser engagement indicator may include values such as 0 (no userengagement is detected) indicative that the second user is not actuatinga user engagement button or actively performing a complex gesture orother indicia of user engagement), or 1 (user engagement is detected).

At step 410, the method can include a step for receiving, from a secondmobile device, a second control instruction from a second mobile device,the second control instruction comprising a second user engagementindicator. The second user engagement indicator may include values suchas 0 (no user engagement is detected) indicative that the second user isnot actuating a user engagement button or actively performing a complexgesture or other indicia of user engagement), or 1 (user engagement isdetected).

At step 415, the method can include a step for determining, based on thefirst user engagement indicator that first user engagement meets athreshold. For example, this step may include determining that the firstuser device has provided a signal indicative that a user engagementfeature is pressed, motioned, gestured, actuated, etc. At step 420, themethod can include a step for determining, based on the second userengagement indicator that second user engagement meets a threshold. Forexample, this step may include determining that the second user devicehas provided a signal indicative that a user engagement feature ispressed, motioned, gestured, actuated, etc.

At step 425, the method can include a step for causing, via theprocessor, a vehicle controller to operate the vehicle to park a trailerpivotably disposed with the vehicle based on the first curvaturecommand, the first user engagement indicator, and the second userengagement indicator, to complete a remote trailer parking operation.This step may include receiving, via the processor, a second curvaturecommand from the second mobile device, and causing the vehiclecontroller to operate the vehicle based on the first and second controlinstructions. In some aspects, the step may further include aggregatingthe first control instruction and the second control instruction, andcausing the vehicle controller to operate the vehicle based on theaggregated first and second control instructions, aggregating the firstcontrol instruction and the second control instruction, and causing thevehicle controller to operate the vehicle based on the aggregated firstand second control instructions.

In one example embodiment, the aggregating step includes determining anengagement consensus for aspects of the control commands received as thefirst control instruction and the second control instruction. The systemmay aggregate the signals by determining, via the processor, that one ofthe first user engagement indicator and the second user engagementindicator meets a third threshold, such as an agreement within thresholdlimits of the curvature command. For example, the aggregation mayinclude averaging the first and second curvature control commands,determining if the aggregated or averaged difference is within apredetermined tolerance to any one or more other control instructions,and stopping the vehicle responsive to determining that the averagedcurvature command is outside of a predetermined tolerance when comparedto the first and second curvature control commands.

In another aspect, this step further includes determining that aposition of the first mobile device meets a device localization rule,determining that a second position associated with a location of thesecond mobile device meets the device localization rule; and causing thevehicle controller to operate based on the device localization rule.This may include localizing one or more users respective to a positionof the vehicle and/or the trailer, localizing one or more obstacles inthe operating environment, and evaluating whether any of the users havean obstructed view of the vehicle as it becomes proximate to one or moreof the obstacles. For example, the step may include evaluating theposition of users and obstacles, and determining, based on dataindicative of the vehicle and trailer dimensions, that one or more usersare unable to see the obstacle in a present location. The system maydetermine a second position at which one or more of the users may bepositioned to alleviate the obstructed view, stop the vehicle, andoutput an instruction via one or more mobile devices indicative of arecommended user position. Responsive to determining that the blind spotis now viewable by one or more users, the step may include causing thevehicle to complete the parking maneuver.

In another aspect, this step may further include determining that athird user may alleviate the obstructed view, and recommend that a thirduser and/or user device be used to complete the parking maneuver. Thesystem may further generate a recommendation of a position for the thirduser based on localization data indicative of a position of theobstruction, the first, user, the second user, the vehicle, and/or thetrailer.

Embodiments described herein can provide ways to integrate a trailerbirds-eye view video feed with existing wiring hardware and controlmodules onboard a towing vehicle without the need to add additionalwiring connections. The disclosed methods and system may be usefulbecause they provide a better user experience for those that may wish toview the trailer at a birds-eye view similar to features in the towingvehicle, and without adding additional wiring connections or an upgradedcontrol module onboard the towing vehicle.

In the above disclosure, reference has been made to the accompanyingdrawings, which form a part hereof, which illustrate specificimplementations in which the present disclosure may be practiced. It isunderstood that other implementations may be utilized, and structuralchanges may be made without departing from the scope of the presentdisclosure. References in the specification to “one embodiment,” “anembodiment,” “an example embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Further, when afeature, structure, or characteristic is described in connection with anembodiment, one skilled in the art will recognize such feature,structure, or characteristic in connection with other embodimentswhether or not explicitly described.

Further, where appropriate, the functions described herein can beperformed in one or more of hardware, software, firmware, digitalcomponents, or analog components. For example, one or more applicationspecific integrated circuits (ASICs) can be programmed to carry out oneor more of the systems and procedures described herein. Certain termsare used throughout the description and claims refer to particularsystem components. As one skilled in the art will appreciate, componentsmay be referred to by different names. This document does not intend todistinguish between components that differ in name, but not function.

It should also be understood that the word “example” as used herein isintended to be non-exclusionary and non-limiting in nature. Moreparticularly, the word “exemplary” as used herein indicates one amongseveral examples, and it should be understood that no undue emphasis orpreference is being directed to the particular example being described.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory (e.g., tangible) medium thatparticipates in providing data (e.g., instructions) that may be read bya computer (e.g., by a processor of a computer). Such a medium may takemany forms, including, but not limited to, non-volatile media andvolatile media. Computing devices may include computer-executableinstructions, where the instructions may be executable by one or morecomputing devices such as those listed above and stored on acomputer-readable medium.

With regard to the processes, systems, methods, heuristics, etc.described herein, it should be understood that, although the steps ofsuch processes, etc. have been described as occurring according to acertain ordered sequence, such processes could be practiced with thedescribed steps performed in an order other than the order describedherein. It further should be understood that certain steps could beperformed simultaneously, that other steps could be added, or thatcertain steps described herein could be omitted. In other words, thedescriptions of processes herein are provided for the purpose ofillustrating various embodiments and should in no way be construed so asto limit the claims.

Accordingly, it is to be understood that the above description isintended to be illustrative and not restrictive. Many embodiments andapplications other than the examples provided would be apparent uponreading the above description. The scope should be determined, not withreference to the above description, but should instead be determinedwith reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled. It is anticipated andintended that future developments will occur in the technologiesdiscussed herein, and that the disclosed systems and methods will beincorporated into such future embodiments. In sum, it should beunderstood that the application is capable of modification andvariation.

All terms used in the claims are intended to be given their ordinarymeanings as understood by those knowledgeable in the technologiesdescribed herein unless an explicit indication to the contrary is madeherein. In particular, use of the singular articles such as “a,” “the,”“said,” etc. should be read to recite one or more of the indicatedelements unless a claim recites an explicit limitation to the contrary.Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments could include, while other embodiments may not include,certain features, elements, and/or steps. Thus, such conditionallanguage is not generally intended to imply that features, elements,and/or steps are in any way required for one or more embodiments.

That which is claimed is:
 1. A method comprising: receiving, via aprocessor, a first control instruction from a first mobile deviceindicative of a first curvature command providing directional control ofa vehicle, and a first user engagement indicator; receiving a secondcontrol instruction from a second mobile device, the second controlinstruction comprising a second user engagement indicator; determining,based on the first user engagement indicator that first user engagementmeets a first threshold; determining, based on the second userengagement indicator that second user engagement meets a secondthreshold; and causing, via the processor, a vehicle controller tooperate the vehicle to park a trailer pivotably disposed with thevehicle based on the first curvature command, the first user engagementindicator, and the second user engagement indicator.
 2. The methodaccording to claim 1, further comprising: receiving, via the processor,a second curvature command from the second mobile device; and causingthe vehicle controller to operate the vehicle based on the first controlinstruction and the second control instruction.
 3. The method accordingto claim 2, wherein causing the vehicle controller to operate thevehicle based on the first control instruction and the second controlinstruction comprises: aggregating the first control instruction and thesecond control instruction; and causing the vehicle controller tooperate the vehicle based on the aggregated first and second controlinstructions.
 4. The method according to claim 3, further comprising:receiving a third control instruction from a third mobile device;determining that the third mobile device is exempt from input to anaggregated control instruction.
 5. The method according to claim 4,where determining that the third mobile device is exempt includesdetermining the third mobile device is exempt based on at least one of alocation of the third mobile device or a determination that a devicemeets an exemption criteria.
 6. The method according to claim 1, furthercomprising: receiving, from the second mobile device, a third controlinstruction, the third control instruction comprising a third userengagement indicator; determining, via the processor, that the thirduser engagement indicator meets a third threshold; and stopping thevehicle.
 7. The method according to claim 2, wherein causing the vehiclecontroller to operate the vehicle is based on the first curvaturecommand being within a threshold difference from the second curvaturecommand.
 8. The method according to claim 1, further comprising:determining that a first location associated with the first mobiledevice meets a device localization rule; determining that a secondposition associated with a second location associated with the secondmobile device meets the device localization rule.
 9. The methodaccording to claim 1, wherein that the first mobile device is anadministrator device, further comprising: receiving, via the processor,information indicative of the second mobile device authorized to controlthe vehicle controller to operate the vehicle based on the aggregatedfirst and second control instructions; and causing, via the processor,the vehicle controller to operate the vehicle to park the trailer basedon the first curvature command, the first user engagement indicator, thesecond user engagement indicator, and the information indicative of thesecond mobile device authorized to control the vehicle controller. 10.The method according to claim 9, further comprising: receiving, from thefirst mobile device, information indicative of a third location to wherethe second mobile device is to be positioned; and sending, to the secondmobile device, information indicative of the third location to where thesecond mobile device is to move, the information configured to bepresented by the second mobile device in a map view.
 11. The methodaccording to claim 10, wherein the information indicative of the thirdlocation was received as user input by the first mobile device.
 12. Themethod according to claim 11, wherein receiving, from the first mobiledevice, information indicative of the third location to where the secondmobile device is to be positioned comprises: receiving informationindicative of a perimeter around the third location that was received asuser input by the first mobile device.
 13. The method according to claim1, wherein the first mobile device is an administrator device, furthercomprising: receiving, from the first mobile device, informationindicative that the second mobile device is authorized to maneuver thevehicle based on a single curvature command; and causing, via theprocessor, the vehicle controller to operate the vehicle to park thetrailer further based on a curvature command from the second mobiledevice.
 14. The method according to claim 1, further comprising:sending, to the first mobile device, information configured to bepresented by the first mobile device by a human-machine interfaceconfigured to receive input for controlling an operation of the vehicle.15. The method according to claim 1, further comprising: sending, to thefirst mobile device, information indicative of a location of the secondmobile device and configured to be presented by the first mobile device.16. The method according claim 1 further comprising: sending, to thefirst mobile device, information indicative of a control input receivedfrom the second mobile device and configured for presentation by thefirst mobile device, wherein the presentation identifies the controlinput as an adjustment to the first control instruction or the secondcontrol instruction.
 17. A system, comprising: a control module disposedin a vehicle; and a memory disposed in the control module, storingexecutable instructions that cause the control module to execute theexecutable instructions to: receive a first control instruction from afirst mobile device indicative of a first curvature command providingdirectional control of the vehicle, and a first user engagementindicator; receive a second control instruction from a second mobiledevice, the second control instruction comprising a second userengagement indicator; determine, based on the first user engagementindicator that first user engagement meets a first threshold; determine,based on the second user engagement indicator that second userengagement meets a second threshold; and cause, via the control module,a vehicle controller to operate the vehicle to park a trailer pivotablydisposed with the vehicle based on the first curvature command, thefirst user engagement indicator, and the second user engagementindicator.
 18. The system according to claim 17, wherein theinstructions are further configured to cause the control module toexecute the executable instructions to: receive, at the control module,a second curvature command from the second mobile device; and cause thevehicle controller to operate the vehicle based on the first controlinstruction and the second control instruction.
 19. The system accordingto claim 17, wherein the instructions are further configured to executethe executable instructions to: aggregate the first control instructionand the second control instruction; and cause the vehicle controller tooperate the vehicle based on the aggregated first and second controlinstructions.
 20. The system according to claim 17, wherein theinstructions are further configured to execute the executableinstructions to: receive a third control instruction from a third mobiledevice; determine that the third mobile device is exempt from input toan aggregated control instruction.
 21. The system according to claim 20,wherein the instructions for determining that the third mobile device isexempt include instruction to determine the third mobile device isexempt based on at least one of a location of the third mobile device ordetermination a device meets an exemption criteria.
 22. The systemaccording to claim 17, wherein the instructions are further configuredto execute the executable instructions to: determine that one of thefirst user engagement indicator and the second user engagement indicatormeets a third threshold; and stop the vehicle.
 23. The system accordingto claim 17, wherein causing the vehicle controller to operate thevehicle is based on the first curvature command being within a thresholddifference from a second curvature command.
 24. The system according toclaim 17, wherein the instructions are further configured to execute theexecutable instructions to: determine that a first location associatedwith the first mobile device meets a device localization rule; determinethat a second position associated with a second location associated withthe second mobile device meets the device localization rule; and causethe vehicle controller to operate based on the device localization rule.25. The system according to claim 17, wherein the first mobile device isan administrator device, wherein the instructions are further configuredto execute the executable instructions to: receive informationindicative of the second mobile device authorized to control the vehiclecontroller to operate the vehicle based on the aggregated first andsecond control instructions; and cause the vehicle controller to operatethe vehicle to park the trailer based on the first curvature command,the first user engagement indicator, the second user engagementindicator, and the information indicative of the second mobile deviceauthorized to control the vehicle controller.
 26. The system accordingto claim 25, wherein the instructions are further configured to executethe executable instructions to: receive, from the first mobile device,information indicative of a third location to where the second mobiledevice is to be positioned; and send, to the second mobile device,information indicative of the third location to where the second mobiledevice is to move, the information configured to be presented by thesecond mobile device in a map view.
 27. The system according to claim26, wherein the information indicative of the third location wasreceived as user input by the first mobile device.
 28. The systemaccording to claim 27, wherein the instructions for receiving, from thefirst mobile device, information indicative of the third location towhere the second mobile device is to be positioned includes instructionsto: receive information indicative of a perimeter about the thirdlocation that was received as user input by the first mobile device. 29.The system according to claim 17, wherein the first mobile device is anadministrator device, the instructions are further configured to executethe executable instructions to: receive, from the first mobile device,information indicative that the second mobile device is authorized tomaneuver the vehicle based on a single curvature command; and cause thevehicle controller to operate the vehicle to park the trailer furtherbased on a curvature command from the second mobile device.
 30. Thesystem according to claim 17, wherein the instructions are furtherconfigured to execute the executable instructions to: send, to the firstmobile device, information configured to be presented by the firstmobile device by a human-machine interface that is configured to receiveinput for controlling an operation of the vehicle.
 31. The systemaccording to claim 17, wherein the instructions are further configuredto execute the executable instructions to: send, to the first mobiledevice, information indicative of a location of the second mobiledevice, the information configured to be presented by the first mobiledevice.
 32. The system according to claim 17, wherein the instructionsare further configured to execute the executable instructions to: send,to the first mobile device, information indicative of a control inputreceived from the second mobile device and configured for presentationby the first mobile device, wherein the presentation identifies thecontrol input as an adjustment to the first control instruction or thesecond control instruction.
 33. A non-transitory computer-readablestorage medium in a vehicle control module, the non-transitorycomputer-readable storage medium having instructions stored thereuponwhich, when executed by the vehicle control module: receive a firstcontrol instruction from a first mobile device indicative of a firstcurvature command providing directional control of a vehicle, and afirst user engagement indicator; receive, from a second mobile device, asecond control instruction from the second mobile device, the secondcontrol instruction comprising a second user engagement indicator;determine, based on the first user engagement indicator that first userengagement meets a first threshold; determine, based on the second userengagement indicator that second user engagement meets a secondthreshold; and cause a vehicle controller to operate the vehicle to parka trailer pivotably disposed with the vehicle based on the firstcurvature command, the first user engagement indicator, and the seconduser engagement indicator.
 34. The non-transitory computer-readablestorage medium according to claim 33, further comprising causing thevehicle control module to: receive a second curvature command from thesecond mobile device; and cause the vehicle controller to operate thevehicle based on the first control instruction and the second controlinstruction.